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5 Abstract 

^ 7 such as the eutrophication of a lake, the collapse of a coral reef, or the end of a glacial period. 



O 

> 

X 



Because such transitions often unfold on temporal and spatial scales that can be difficult to 



O 

^ 9 approach by experimental manipulation, research has often relied on historical observations as a 



r— n 10 source of natural experiments. Here we examine a critical difference between selecting systems 

Ph 11 for study based on the fact that we have observed a critical transition and those systems for 

• 2 12 which we wish to forecast the approach of a transition. This difference arises by conditionally 

qh 13 selecting systems known to experience a transition of some sort and failing to account for the 

14 bias this introduces - a statistical error often known as the Prosecutor's Fallacy. By analysing 

> 1. simulated systems that have experienced transitions purely by chance, we reveal an elevated rate 

^ 16 of false positives in common warning signal statistics. We further demonstrate a model-based 

17 approach that is less subject to this bias than these more commonly used summary statistics. 

18 We note that experimental studies with replicates avoid this pitfall entirely. 

19 Keywords: early warning signals, tipping point, alternative stable states, likelihood methods 



20 1. Introduction 

21 Mathematics . . . while assisting the trier of fact in the search of truth, must not cast 

22 a spell over him. - California Supreme court, 1968. 

23 In the case of People v. Collins 1968, California Supreme Court considered the evidence of an 

24 expert witness described by the court as "an instructor of mathematics at a state college" , which 

25 concluded that the probability that a randomly selected individual would match the description 



26 given by the victim would be less than 1 in 12 million (Supreme Court, 1968). The prosecution 
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27 had produced an individual matching the prosecutor's detailed description, and convinced by 

28 the mathematics, the lower courts had found him guilty. 

29 The prosecution has only observed that the probability of seeing the evidence (E) they 

30 produced given a random innocent individual (/), P{E\I) is very small. From this one cannot 

31 conclude that the individual is indeed guilty, that is, that the probability the individual is 

32 innocent given the evidence P{I\E) is also very small. In a city with millions of people, there 

33 might be several individuals who match the description of the evidence. Mathematically P{E\I) 

34 need not equal P{I\E), instead, these expressions are related by Bayes theorem. 



P{E\I) = P{I\E) 



P{E) 



P{E) ^ 1 and P{I) ~ 1 , so P{E\I) ^ P{I\E)P{E), and consequently we cannot conclude 
that P{I\E) ^ 1 from P{E\I) ^ 1. Realizing this mistake, the California Supreme Court 
reversed the decision, and the case became a widely recognized example of the Prosecutor's 



Fallacy (Thompson and Schumann, 1987). Here we explore how a similar misconception can 
arise from the use of historical data to evaluate methods for detecting early warning signals of 
critical transitions. 

Catastrophic transitions or tipping points, where a complex system shifts suddenly from one 
state to another, have been implicated in a wide array of ecological and global climate systems 



ipuch as lake ecosystems (Carpenter, 2011), coral reefs (Mumby et al. , 2007), savannah (Kefi 



et al. , 2007), fisheries (Berkes et al. , 2006), and tropical forests (Hirota et al. , 2011). Re- 
cent research has begun to identify statistical patterns commonly associated with these sudden 
catastrophic transitions which could be used as an early warning sign to identify an approaching 
tipping point, which might provide managers time to react to and avert an undesirable state 



shift (Scheffer et al. , 2009 Lenton, 2011). An array of statistical patterns associated with tip- 
ping point phenomena has been suggested for the detection of early warning signals associated 
with such sudden transitions. Two of the most commonly used are a pattern of increasing 



variance (Carpenter and Brock, 2006) and a pattern of increasing autocorrelation (van Nes and 



Scheffer 2007), which have been tested in both experimental manipulation (Drake and Griffen 



^010 Carpenter 2011 Veraart et al. 2011 Dai et al. 2012) and historical observations (Livina 



find LentonI 2007 


Dakos et al. 


2008; 


Lenton et al. 


2012; 


Ditlevsen and Johnsen, 


2010 


Guttal 
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55 and Jayaprakash 2008 Thompson and Sieber 2010). 



Testing patterns on historical data 

Historical examples of sudden transitions taken from the paleo-climate record provide an 
important way to test and evaluate potential leading indicator methods, and have been widely 



2007 


Dakos et al. 


2008 


Lenton et al. 


2012 


Ditlevsen 



and Johnsen 2010 Guttal and Jayaprakash 2008 Thompson and Sieber 2010). Similarly, it 



has been suggested that data gathered from ecological systems such as lakes that were monitored 
before they experienced sudden eutrophication, or grasslands subjected to overgrazing, could 



ipontain data that could help reveal when similar systems are approaching a tipping point ( Car 



penter|[2011j). 

However, testing methods for early warning signals against historical examples of transitions 
is susceptible to statistical mistakes that arise from selecting data conditional on that data 
having already exhibited a sudden transition. A central tenant of early warning theory is that 
the system in question is slowly approaching a tipping point that lies some unknown distance 
away. If nothing is done to remedy the situation, this slow change will inevitably carry the 
system beyond the tipping point, which introduces a sudden, rapid transition into an undesirable 



state (Scheffer et al. , 2009). This process can be described mathematically as a bifurcation, in 
which a slowly changing parameter reaches a critical value that causes the system stability to 
change. 

Not all sudden transitions are caused by some "guilty" process slowly driving the system over 
a tipping point - the kind of process that early warning signals are designed to detect. Some 
systems may experience such transitions purely by chance, leaving a stable state on an extremely 



ljunlikely excursion that happens to stray to far from the stable attractor {e.g. Ditlevsen and 
|Johnsen| |2010 Lenton[ 2011 consider this possibility in transitions that arise from analyzing 
historical climate record). Like the evidence presented before the California Supreme Court in 
1968, the chance of observing such an "innocent" transition a priori may be very small, but when 
selected from a historical record of many possible transitions, this possibility can no longer be 
ignored. 

Figure 1 shows a schematic illustrating critical transitions under each of these scenarios. In 
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84 the left panel, the system experiences a bifurcation and should contain an early warning signal. 

85 In the right panel, a similar-looking trajectory emerges from a simulation of a stable system 

86 which should not contain a warning signal. While the simulation of the bifurcation scenario 

87 shown on the left produces a similar transition every time, the transition shown on the right is 

88 somewhat less likely, occurring in only 1% of simulations. 

89 [Figure 1 about here.] 



90 2. Methods and Results 

91 To investigate if early warning signals are vulnerable to this fallacy, we simulate a system that 

92 is not driven towards a bifurcation such as in Fig reffig:l(b). This simulation approach allows 

93 us to determine whether examining historical events is a valid way to test the utility of these 

94 indicators. We simulated 20,000 replicates of a stochastic individual-based birth-death process 



95 with an Allee threshold (Courchamp et al. , 2008), which arises from positive fitness effects at 



96 low densities. Above the Allee threshold the population returns to a positive equilibrium size, 

97 whereas below the threshold the population decreases to zero. The model can be represented as 

98 a continuous time birth-death process where births and deaths are Poisson events which depend 

99 on the current density with rates given by 



d{n) = en + a, (3) 

100 a model with a linear death rate and density-dependent birth rate that drives the Allee 

101 effect at low densities and limits growth at high densities. In this model n indicates the discrete 

102 number of individuals in the population, K indicates a carrying capacity as set by a limiting 

103 resource, e a per-capita death rate (the e scaling term in the birth equation allows the carrying 

104 capacity K to correspond to a positive equilibrium point), a an additional mortality imposed 

105 on the population such as harvest, h is a parameter controlling at what population size the 

106 addition of more individuals switches from conferring a positive benefit on growth from Allee 

107 interactions n < h to a. negative impact on growth due to increased competition, n > h. The 



108 key feature of this model is the alternate stable states introduced by this effect; other functional 

109 forms for Eq. ^ could serve equally well for these simulations (see e.g. Scheffer et al. , 2001). 
no Though this system can be forced through a bifurcation by increasing the death rate, in these 

111 simulations all parameters are held constant and no bifurcation occurs. Consequently we do not 

112 anticipate an early warning signal of an approaching bifurcation. 

113 The simulation starts from the positive equilibrium population size. Though the chance of 

114 a transition across the AUee threshold in any given time step is small, given enough time this 

115 system will eventually experience such a rare event driving the population extinct. We ran each 

116 replicate over 50,000 time units, sampling the system every 50 time units. In this time window 

117 266 of the 1,000 replicates experience population collapse. To keep the examples of comparable 

118 sample size, we focus on a section of the data 500 time points prior to the system approaching 

119 the transition. 

120 To test whether selecting systems that have experienced spontaneous transitions could bias 

121 the analysis towards false positive detection of early warning signals, (the Prosecutor's Fallacy) 

122 we selected replicates conditional on having collapsed in the simulations. We then selected a 

123 window around each system that ended just before the collapse, while the population values 

124 were still above the Allee threshold. For each replicate, we calculated the most common early 



125 warning indicators, variance and autocorrelation {e.g. Carpenter and Brock 2006; Dakos et al. 



2008 Scheffer et al. , 2009), around a moving window equal to half the length of that time series. 



127 To test for the presence of a warning signal in these indicators we computed values of 

128 Kendall's r for both indicators for each of the 266 replicates. Kendall's r is a non-parametric 

129 measure of rank correlation frequently used to identify an increasing trend (r > 0) in early 

2011), defined as T^n{n — 1) in n observations. r takes 



130 warning signals (Dakos et al. 



2008 



131 values in (—1, 1). The distribution of r values observed across these replicates is shown in Fig- 

132 ure [2] We compare the distribution of r from all the simulations to the distribution conditioned 

133 on experiencing a chance transition to the alternative stable state. To avoid an effect of sample 

134 size the time series are all chosen to be the same length. 



pair of observations {xi,yi) and {xj,yj) are concordant if Xi > Xj and yi > yj or Xi < Xj and yi < yj and 
discordant otherwise; equalities excepted. 



5 



135 To demonstrate the effect we observe is not unique to models with Allee effects, we provide 

136 an example of the effect arising in a discrete-time model with two non-zero stable states adapted 



137 from (May, 1977), 




(4) 



138 which combines a logistic growth model with a saturating predator response (See May (1977) 

139 for detailed discussion), shown in Figure [3] Code to replicate the analysis can be found at 

140 https : //github . com/ cboettig/earlywarning/tree/prosecutor/. 

141 [Figure 2 about here.] 

142 [Figure 3 about here.] 

143 For each of these replicates we also take a model-based approach, estimating parameters for 

144 an approximate linear model of the system approaching a saddle node bifurcation, as described 



145 by Boettiger and Hastings (2012), 



dX = ^Mn) - Xt)dt + a^Mn)dBt (5) 

146 In this model, the parameter m describes the approach towards the saddle-node bifurcation. 

147 Estimates m < are expected in systems approaching a bifurcation, while for stable systems m 

148 should be approximately zero. None of the estimates across the 266 simulations differed from 

149 zero in our study, hence the model-based estimation shows no evidence of bias on data that has 

150 been selected conditional on collapse. 

151 3. Discussion 

152 The attempts to detect early warning signs for critical transitions are based on the concept 



153 of a deteriorating environment as embodied in a changing parameter Scheffer et al. (2009), 

154 which is a different kind of transition than one which is driven instead by stochasticity in an 

155 environment which is otherwise constant and exhibiting no directional change. When trying 

156 to use historical data to understand critical transitions we often do not know which category, 

157 changing environment or simply chance, an observed large change falls into. 
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158 We have shown here that systems which undergo rare sudden transitions due to chance look 

159 statistically different from their counterparts that do not, even though they are driven by the 

160 same stochastic process. In particular, such conditionally selected examples are more likely to 

161 show signs associated with an early warning of an approaching tipping point, such as increasing 

162 variance or increasing autocorrelation, as measured by Kendall's r. This increases the risk of 

163 false positives - cases in which a warning signal being tested appears to have successfully detected 

164 an underlying change in the system leading to a tipping point, when in fact the example comes 

165 instead from a stable system with no underlying change in parameters. Figure 2 shows that 

166 many of the chance crashes show values of r that are significantly larger than those observed in 

167 the otherwise identical replicates that did not experience a chance transition, thus "detecting" 

168 an underlying change in the system dynamics that is not in fact present. 

169 3.1. Chance transitions are false positives for early warning signals 

170 It seems tempting to argue that this bias towards positive detection in historical examples 

171 is not problematic - each of these systems did indeed collapse, so the increased probability of 

172 exhibiting warning signals could be taken as a successful detection. Unfortunately this is not 

173 the case. At the moment the forecast is made, these systems are not likely to transition, since 

174 they experience a strong pull towards the original stable state. A closer look at the patterns 

175 involved shows why common indicators such as autocorrelation and variance can be misleading. 

176 As the system gets farther from its stable point, it it more likely to draw a random step that 

177 returns it towards the stable point. Despite this, there is always some probability that it will 

178 move further still, so systems that do cross the tipping point must do so rather quickly by a 

179 string of events. This pattern, clearly visible before the crashes in each of the examples in Figure 

180 1, produces a string of observations that appear more highly autocorrclatcd (if wc arc sampling 

181 the system frequently enough to catch the excursion at all) than we observe in the rest of the 

182 fluctuations around the equilibrium. Yet this autocorrelation comes from a chance trajectory 

183 moving quickly away from the stable state, not from the critical slowing down pattern in the 

184 return times to the stable state which precede a saddle-node bifurcation and motivate the early 

185 warning signal. 

186 This longer than expected excursion results in a higher than expected variance in that window 
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187 as well. Both variance and autocorrelation are calculated using a moving window over the time- 

188 series, which allows the method to pick out a pattern of change as the window moves along the 

189 sequence. If this chance excursion that precedes the crash happens to fill a significant part of 

190 the moving window, the resulting pattern will tend to show an increase in autocorrelation or 

191 variance. If the chance excursion is relatively rapid compared to the frequency at which the 

192 system is observed (spacing of the data) or the width of the moving window, the excursion may 

193 not significantly alter the general pattern. In this way, some of the events in which a crash is 

194 observed will appear to present these statistical patterns of increased variance or autocorrelation 

195 without being harbingers of approaching critical transitions. 

196 3.2. The truncation of observations 

197 If we had a complete knowledge of the system dynamics, then we could eliminate the bias 

198 we observe here since the bias arises from the transient branch of the trajectory that crosses 

199 the threshold, and if the system were truncated at the minimum of the potential then the 

200 effect we emphasize here would not appear. But, it is not possible to truncate the system in 

201 any practical application. The precise location of the minimum of the potential which is the 

202 location of the deterministic equilibrium is unknown. Moreover, under the hypothesis that the 

203 system is approaching a critical transition, the location of the minimum potential moves so it 

204 cannot easily be estimated by previous observations, (see Figure Ic where the equilibrium point 

205 moves in the direction of the transition). Thus it is neither practical nor desirable to suggest 

206 that historical time series can be used by following a simple truncation rule that avoids the 

207 branch of a trajectory crossing the threshold to another basin of attraction. Exactly where a 

208 particular study will choose to truncate such a trajectory will necessarily be arbitrary without an 

209 underlying model of the process. Frequently this is done by removing the very steep, monotonic 

210 branch of the trajectory expected once the system crosses the unstable threshold. Such an 

211 approach corresponds with our choice of termination and produces the bias we discuss here. 

212 The examples of Figure 1, though only single replicates, may be useful in illustrating these 

213 issues. Figure Ic, top panel shows a sample trajectory of a system with a parameter shift, while 

214 lb shows a trajectory without a shift. Both trajectories become more highly autocorrelated and 

215 higher variance near the end of the time series (time increases on the y axis in Figure 1). The 
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216 part of the time series following the critical transition shows a fast and monotonic trajectory 

217 to the unstable trajectory, and would usually be excluded by an analysis for warning signals in 

218 advance of the transition. No such clear pattern exists prior to the transition in Figure lb. An 

219 alternative proposal to terminate the trajectory in panel B earlier would also risk decreasing the 

220 signal seen in panel c, and would be inconsistent with the application of warning signals in the 

221 forecasting context, where there would be no such truncation. 

222 3.3. Comparing to the model-based method 

223 In our numerical experiment, the model-based estimate of early warning signals appears more 

224 robust than the summary statistics, producing the same estimates on both the conditionally 

225 selected replicates as on a random sample of the replicates. This is a consequence of the more 

226 rigid specifications that come with a model-based approach - the pattern expected is less general 

227 than any increase in variance or autocorrelation, but instead must be one that matches its 

228 approximation of the saddle-node bifurcation. This observation highlights the difference between 

229 the pattern driving the false positive trends in increasing variance and increasing autocorrelation 

230 and the pattern anticipated in the saddle-node model. This should not however be taken as 

231 evidence that the model-based approach is immune to the bias of the Prosecutor's Fallacy. 

232 3.4. Importance of experimental approaches 

233 The problem we highlight ultimately stems from the difficulty of having only a single re- 

234 alization with which to examine a complex problem. The only way to deal with this problem 

235 embodied is through replication, as can be done in an experimental system in laboratory ma- 



236 nipulations such as Drake and Griffen (2010); Veraart et al. (2011); Dai et al. (2012) and at the 



237 scale of whole lake ecosystems in Carpenter (2011). Experimental procedures avoid the hazard 



238 of the Prosecutor's fallacy by generating a complete sample of replicates, rather than selecting 

239 a subset of cases from some larger historical sample. 
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(a) Birth rate and death rate functions (b) Trajectory of a chance (c) Trajectory during an 
used in the model transition from a stable system underlying parameter shift 




state State State 



Figure 1: The Prosecutor's Fallacy, (a) Plot of the model functions shown in Eq ^ with parameters a — 180, 
K = 500, e = .5, and h = 200. When the death rate is higher than the birth rate, the system dynamics drive 
the state (population size) to smaller values. When birth rate is higher, the system moves right, as indicated 
by the arrows, (b) The potential energy is given by the negative integral of b{n) — d(n), shown in the lower 
plot. The potential function gives an intuitive picture of the stability of a system by imagining the curve as 
a surface on which a ball is free to bounce across - wells correspond to stable points and peaks to unstable 
points. While most trajectories remain near the stable well, some transition out merely by chance. An example 
of such a trajectory is shown in the top panel, in which time increases along the vertical axis. Though initially 
oscillating around the stable state, a chance excursion carries it beyond the Alice threshold (vertical dotted line) . 
Such chance trajectories can produce the statistical patterns as observed in true critical transitions seen in panel 
(c): Early warning signals are aimed at detecting systems which are slowly moving towards a tipping point or 
bifurcation, illustrated in the successive curves (deteriorating and critical). Top panel: An example trajectory 
from a simulation under this process shows the state of the system as the potential moves towards the bifurcation 
point. The original position of the AUee threshold is shown by the vertical dotted line (though it moves slightly 
as the parameter changes). 
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Figure 2: The distribution of the correlation statistic r for two early warning indicators (variance, autocorrelation) 
on replicates conditionally selected for having collapsed by chance in simulations is shown in grey bars. Solid 
lines indicate the estimated density of the statistic from a random sample of the simulations (not conditional 
on observing a transition). Positive values of r correspond to a pattern of an indicator increasing with time; 
typically taken as evidence that a system is approaching a critical transition. In these simulations, the pattern 
arises instead from the Prosecutor's fallacy of conditional selection. 
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Figure 3: The identical analysis from Figure [5] is shown for the model in Eq Q using parameters r = 0.75, 
K = 10, a = 1.7, Q = 3, and H = 1. A similar statistical bias, particularly towards positive values of tau occurs 
in this model as well. 
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